# Switching from the in-tree RBD driver to CSI (Ceph CSI)

The [rbd-in-tree-to-ceph-csi-migration-helper.sh](https://github.com/deckhouse/deckhouse/blob/main/modules/031-ceph-csi/tools/rbd-in-tree-to-ceph-csi-migration-helper.sh) script was created to simplify the migration process.
Before running it, delete the Pod (scale the StatefulSet/Deployment down to zero) which uses the PVC. You will have to manually run a command in the Ceph cluster to rename the RBD image (since Ceph CSI uses a different name format) during the migration.

**Caution!** It is assumed that the `ceph-csi` module is enabled and configured and that the old driver continues to work.

The script will back up the manifests if the PVCs and PVs to be migrated, delete the old manifests and create the new ones. Note that deleting the PV will not cause the RBD image in the Ceph cluster to be deleted, since the script will rename it beforehand.

The script requires the PVCs and PVs to work; it will use their manifests to obtain the parameters specific to Ceph CSI. You can use the following manifest to create them:

```yaml
kubectl create -f - <<"END"
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: sample
  namespace: d8-monitoring
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: ceph-csi-rbd
END
```

Below is an example of the output generated by the script:

```bash
root@kube-master-0:~# ./rbd-in-tree-to-ceph-csi-migration-helper.sh default/sample default/data-test-0
Rename the rbd image in your ceph cluster using the following command:
>rbd mv kube/kubernetes-dynamic-pvc-162a2c43-568e-40ab-aedb-a4632a613ecd kube/csi-vol-162a2c43-568e-40ab-aedb-a4632a613ecd
After renaming, enter yes to confirm: yes
PersistentVolumeClaim data-test-0 and PersistentVolume pvc-4a77a995-ce1e-463c-9726-d05966d3c5ef will be removed (Type yes to confirm): yes
>kubectl -n default delete pvc data-test-0
persistentvolumeclaim "data-test-0" deleted
>kubectl delete pv pvc-4a77a995-ce1e-463c-9726-d05966d3c5ef
persistentvolume "pvc-4a77a995-ce1e-463c-9726-d05966d3c5ef" deleted
>kubectl create -f - <<"END"
{
  "apiVersion": "v1",
  "kind": "PersistentVolumeClaim",
  "metadata": {
    "annotations": {
      "pv.kubernetes.io/bind-completed": "yes",
      "pv.kubernetes.io/bound-by-controller": "yes",
      "volume.beta.kubernetes.io/storage-provisioner": "rbd.csi.ceph.com"
    },
    "finalizers": [
      "kubernetes.io/pvc-protection"
    ],
    "labels": {
      "app": "test"
    },
    "name": "data-test-0",
    "namespace": "default"
  },
  "spec": {
    "accessModes": [
      "ReadWriteOnce"
    ],
    "resources": {
      "requests": {
        "storage": "1Gi"
      }
    },
    "storageClassName": "ceph-csi-rbd",
    "volumeMode": "Filesystem",
    "volumeName": "pvc-4a77a995-ce1e-463c-9726-d05966d3c5ef"
  }
}
END
Apply this manifest in the cluster? (Type yes to confirm): yes
persistentvolumeclaim/data-test-0 created
>kubectl create -f - <<"END"
{
  "apiVersion": "v1",
  "kind": "PersistentVolume",
  "metadata": {
    "annotations": {
      "pv.kubernetes.io/provisioned-by": "rbd.csi.ceph.com",
      "volume.kubernetes.io/provisioner-deletion-secret-name": "csi-new",
      "volume.kubernetes.io/provisioner-deletion-secret-namespace": "d8-ceph-csi"
    },
    "finalizers": [
      "kubernetes.io/pv-protection"
    ],
    "name": "pvc-4a77a995-ce1e-463c-9726-d05966d3c5ef"
  },
  "spec": {
    "accessModes": [
      "ReadWriteOnce"
    ],
    "capacity": {
      "storage": "1Gi"
    },
    "claimRef": {
      "apiVersion": "v1",
      "kind": "PersistentVolumeClaim",
      "name": "data-test-0",
      "namespace": "default",
      "resourceVersion": "14908531",
      "uid": "0ac58d43-75f9-4481-96fd-dcf8ca60ad85"
    },
    "mountOptions": [
      "discard"
    ],
    "persistentVolumeReclaimPolicy": "Retain",
    "storageClassName": "ceph-csi-rbd",
    "volumeMode": "Filesystem",
    "csi": {
      "controllerExpandSecretRef": {
        "name": "csi-new",
        "namespace": "d8-ceph-csi"
      },
      "driver": "rbd.csi.ceph.com",
      "fsType": "ext4",
      "nodeStageSecretRef": {
        "name": "csi-new",
        "namespace": "d8-ceph-csi"
      },
      "volumeAttributes": {
        "clusterID": "60f356ee-7c2d-4556-81be-c24b34a30b2a",
        "imageFeatures": "layering",
        "imageName": "csi-vol-162a2c43-568e-40ab-aedb-a4632a613ecd",
        "journalPool": "kube",
        "pool": "kube",
        "storage.kubernetes.io/csiProvisionerIdentity": "1666697721019-8081-rbd.csi.ceph.com"
      },
      "volumeHandle": "0001-0024-60f356ee-7c2d-4556-81be-c24b34a30b2a-0000000000000005-162a2c43-568e-40ab-aedb-a4632a613ecd"
    }
  }
}
END
Apply this manifest in the cluster? (Type yes to confirm): yes
persistentvolume/pvc-4a77a995-ce1e-463c-9726-d05966d3c5ef created
```

**Caution!** Before switching to containerd, make sure that no log collectors other than log-shipper are used in the cluster. If there are any, you will either have to discard them in favor of the [log-shipper](https://deckhouse.io/documentation/v1/modules/460-log-shipper/) module or reconfigure them to work with containerd. This is because containerd has a different log format and stores log files under a different path.

| CRI        | Log format  | Log files path         |
| ---------- | ----------- | -----------------------|
| Docker     | JSON        | `/var/log/containers/` |
| Containerd | Plain Text  | `/var/log/pods/`       |

## Additional information about the migration process

### Table of contents

- [Switching from the in-tree RBD driver to CSI (Ceph CSI)](#switching-from-the-in-tree-rbd-driver-to-csi-ceph-csi)
  - [Additional information about the migration process](#additional-information-about-the-migration-process)
    - [Table of contents](#table-of-contents)
    - [Manifests of the PVCs and PVs to be migrated](#manifests-of-the-pvcs-and-pvs-to-be-migrated)
    - [PVC and PV manifests to use for importing the specific Ceph CSI parameters](#pvc-and-pv-manifests-to-use-for-importing-the-specific-ceph-csi-parameters)
    - [Renaming an RBD image in a ceph cluster](#renaming-an-rbd-image-in-a-ceph-cluster)
    - [Deleting PVCs and PVs from the cluster](#deleting-pvcs-and-pvs-from-the-cluster)
    - [Generating a new PVC manifest and creating an object in the cluster](#generating-a-new-pvc-manifest-and-creating-an-object-in-the-cluster)
    - [Generating a new PV manifest and creating an object in the cluster](#generating-a-new-pv-manifest-and-creating-an-object-in-the-cluster)

### Manifests of the PVCs and PVs to be migrated

```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/rbd
  creationTimestamp: "2022-11-03T13:15:43Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app: test
  name: data-test-0
  namespace: default
  resourceVersion: "8956688"
  uid: cd6f7b26-d768-4cab-88a4-baca5b242cc5
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: rbd
  volumeMode: Filesystem
  volumeName: pvc-cd6f7b26-d768-4cab-88a4-baca5b242cc5
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1Gi
  phase: Bound
```

```yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    kubernetes.io/createdby: rbd-dynamic-provisioner
    pv.kubernetes.io/bound-by-controller: "yes"
    pv.kubernetes.io/provisioned-by: kubernetes.io/rbd
  creationTimestamp: "2022-11-03T13:15:49Z"
  finalizers:
  - kubernetes.io/pv-protection
  name: pvc-cd6f7b26-d768-4cab-88a4-baca5b242cc5
  resourceVersion: "8956671"
  uid: 4ab7fcf4-e8db-426e-a7aa-f5380ef857c7
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: data-test-0
    namespace: default
    resourceVersion: "8956643"
    uid: cd6f7b26-d768-4cab-88a4-baca5b242cc5
  mountOptions:
  - discard
  persistentVolumeReclaimPolicy: Delete
  rbd:
    image: kubernetes-dynamic-pvc-f32fea79-d658-4ab1-967a-fb6e8f930dec
    keyring: /etc/ceph/keyring
    monitors:
    - 192.168.4.215:6789
    pool: kube
    secretRef:
      name: ceph-secret
    user: kube
  storageClassName: rbd
  volumeMode: Filesystem
status:
  phase: Bound
```

### PVC and PV manifests to use for importing the specific Ceph CSI parameters

The StorageClass created by the ceph-csi module is used.

```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: rbd.csi.ceph.com
  creationTimestamp: "2022-11-03T12:46:20Z"
  finalizers:
  - kubernetes.io/pvc-protection
  name: sample
  namespace: default
  resourceVersion: "8950577"
  uid: abdbb7ea-5da6-47f3-8b76-b968a93b7bc1
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: ceph-csi-rbd
  volumeMode: Filesystem
  volumeName: pvc-abdbb7ea-5da6-47f3-8b76-b968a93b7bc1
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1Gi
  phase: Bound
```

```yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: rbd.csi.ceph.com
    volume.kubernetes.io/provisioner-deletion-secret-name: csi-new
    volume.kubernetes.io/provisioner-deletion-secret-namespace: d8-ceph-csi
  creationTimestamp: "2022-11-03T12:46:27Z"
  finalizers:
  - kubernetes.io/pv-protection
  name: pvc-abdbb7ea-5da6-47f3-8b76-b968a93b7bc1
  resourceVersion: "8950562"
  uid: 6200ce15-b6f2-45af-94d0-828913e850d0
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: sample
    namespace: default
    resourceVersion: "8950550"
    uid: abdbb7ea-5da6-47f3-8b76-b968a93b7bc1
  csi:
    controllerExpandSecretRef:
      name: csi-new
      namespace: d8-ceph-csi
    driver: rbd.csi.ceph.com
    fsType: ext4
    nodeStageSecretRef:
      name: csi-new
      namespace: d8-ceph-csi
    volumeAttributes:
      clusterID: 60f356ee-7c2d-4556-81be-c24b34a30b2a
      imageFeatures: layering
      imageName: csi-vol-880ec27e-5b75-11ed-a252-fa163ee74632
      journalPool: kube
      pool: kube
      storage.kubernetes.io/csiProvisionerIdentity: 1666697721019-8081-rbd.csi.ceph.com
    volumeHandle: 0001-0024-60f356ee-7c2d-4556-81be-c24b34a30b2a-0000000000000005-880ec27e-5b75-11ed-a252-fa163ee74632
  mountOptions:
  - discard
  persistentVolumeReclaimPolicy: Delete
  storageClassName: ceph-csi-rbd
  volumeMode: Filesystem
status:
  phase: Bound
```

### Renaming an RBD image in a ceph cluster

Renaming is mandatory because the Ceph CSI driver uses a different RBD image naming format.

The command is supposed to be run in the Ceph cluster:

```shell
rbd mv kube/kubernetes-dynamic-pvc-<rbd-image-uid> kube/csi-vol-<rbd-image-uid>
```

* `kube` is the name of the pool in the Ceph cluster;
* `kubernetes-dynamic-pvc-<uid>` is the RBD image name format used by the in-tree driver;
* `csi-vol-<uid>` is the RBD image name format used by Ceph CSI.

### Deleting PVCs and PVs from the cluster

```bash
kubectl -n default delete pvc data-test-0
kubectl delete pv pvc-cd6f7b26-d768-4cab-88a4-baca5b242cc5
```

Since the RBD image in the Ceph cluster was renamed in the previous step, deleting PersistentVolume will not cause the image to be deleted.

### Generating a new PVC manifest and creating an object in the cluster

Here is the original manifest with comments:

```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/rbd # replace the annotation with a similar one from the PVC sample
  creationTimestamp: "2022-11-03T13:15:43Z"  # delete
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app: test
  name: data-test-0
  namespace: default
  resourceVersion: "8956688"  # delete
  uid: cd6f7b26-d768-4cab-88a4-baca5b242cc5 # delete
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: rbd
  volumeMode: Filesystem
  volumeName: pvc-cd6f7b26-d768-4cab-88a4-baca5b242cc5
status: # delete
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1Gi
  phase: Bound
```

You will end up with the following manifest:

```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: rbd.csi.ceph.com
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app: test
  name: data-test-0
  namespace: default
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: ceph-csi-rbd
  volumeMode: Filesystem
  volumeName: pvc-cd6f7b26-d768-4cab-88a4-baca5b242cc5
```

Let's create an object in the cluster using this manifest.

### Generating a new PV manifest and creating an object in the cluster

Here is the original manifest with comments:

```yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    kubernetes.io/createdby: rbd-dynamic-provisioner
    pv.kubernetes.io/bound-by-controller: "yes"
    pv.kubernetes.io/provisioned-by: kubernetes.io/rbd
  creationTimestamp: "2022-11-03T13:15:49Z" # delete
  finalizers:
  - kubernetes.io/pv-protection
  name: pvc-cd6f7b26-d768-4cab-88a4-baca5b242cc5
  resourceVersion: "8956671" # delete
  uid: 4ab7fcf4-e8db-426e-a7aa-f5380ef857c7 # delete
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: data-test-0
    namespace: default
    resourceVersion: "8956643" # delete
    uid: cd6f7b26-d768-4cab-88a4-baca5b242cc5 # replace with a new one from the PVC created in the previous step
  mountOptions:
  - discard
  persistentVolumeReclaimPolicy: Delete
  rbd: # delete
    image: kubernetes-dynamic-pvc-f32fea79-d658-4ab1-967a-fb6e8f930dec
    keyring: /etc/ceph/keyring
    monitors:
    - 192.168.4.215:6789
    pool: kube
    secretRef:
      name: ceph-secret
    user: kube
  storageClassName: rbd # replace with ceph-csi-rbd
  volumeMode: Filesystem
  # add the csi section
status: # delete
  phase: Bound
```

The `spec.csi` sample can be borrowed from the PV (sample) created earlier:

```yaml
  csi:
    controllerExpandSecretRef:
      name: csi-new
      namespace: d8-ceph-csi
    driver: rbd.csi.ceph.com
    fsType: ext4
    nodeStageSecretRef:
      name: csi-new
      namespace: d8-ceph-csi
    volumeAttributes:
      clusterID: 60f356ee-7c2d-4556-81be-c24b34a30b2a
      imageFeatures: layering
      imageName: csi-vol-880ec27e-5b75-11ed-a252-fa163ee74632 # replace the uid
      journalPool: kube
      pool: kube
      storage.kubernetes.io/csiProvisionerIdentity: 1666697721019-8081-rbd.csi.ceph.com
    volumeHandle: 0001-0024-60f356ee-7c2d-4556-81be-c24b34a30b2a-0000000000000005-880ec27e-5b75-11ed-a252-fa163ee74632 # replace the uid
```

In the `imageName` and `volumeHandle` fields, replace the uid of the rbd image.

In the sample below, the uid is highlighted with the <uid> tags (`<uid>here<uid>`):

```yaml
imageName: csi-vol-<uid>880ec27e-5b75-11ed-a252-fa163ee74632<uid>
volumeHandle: 0001-0024-60f356ee-7c2d-4556-81be-c24b34a30b2a-0000000000000005-<uid>880ec27e-5b75-11ed-a252-fa163ee74632<uid>
```

You will end up with the following manifest:

```yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: rbd.csi.ceph.com
    volume.kubernetes.io/provisioner-deletion-secret-name: csi-new
    volume.kubernetes.io/provisioner-deletion-secret-namespace: d8-ceph-csi
  finalizers:
  - kubernetes.io/pv-protection
  name: pvc-cd6f7b26-d768-4cab-88a4-baca5b242cc5
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: data-test-0
    namespace: default
    uid: cd6f7b26-d768-4cab-88a4-baca5b242cc7
  mountOptions:
  - discard
  persistentVolumeReclaimPolicy: Delete
  csi:
    controllerExpandSecretRef:
      name: csi-new
      namespace: d8-ceph-csi
    driver: rbd.csi.ceph.com
    fsType: ext4
    nodeStageSecretRef:
      name: csi-new
      namespace: d8-ceph-csi
    volumeAttributes:
      clusterID: 60f356ee-7c2d-4556-81be-c24b34a30b2a
      imageFeatures: layering
      imageName: csi-vol-f32fea79-d658-4ab1-967a-fb6e8f930dec
      journalPool: kube
      pool: kube
      storage.kubernetes.io/csiProvisionerIdentity: 1666697721019-8081-rbd.csi.ceph.com
  volumeHandle: 0001-0024-60f356ee-7c2d-4556-81be-c24b34a30b2a-0000000000000005-f32fea79-d658-4ab1-967a-fb6e8f930dec
  storageClassName: ceph-csi-rbd
  volumeMode: Filesystem
```

Create an object in the cluster using this manifest.

This concludes the migration process.
